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USE OF MATRIX SAMPLING PROCEDURES TO ASSESS ACHIEVEMENT 
IN SOLVING OPEN ADDITION AND SUBTRACTION SENTENCES 

Margarlete A. Montague 
The University of Wyoming 
Laramie « Wyoming 82070 

ABSTRACT 

This study Investigated the feasibility of concurrently and randomly 
sampling examinees and Items In order to estimate group achievement. 

Seven 32 -item tests reflecting a 64^ *ltem universe of simple open 
sentences were used su<d> that item selection (random* systematic) and 
assignment (random* systematic) of Items (four* eight* sixteen) to forms 
were varied. TWenty-four second or third grade populations ware randomly 
selected. Analysis of Variance was used to examine the data. 

Nonsignificant differences were observed with respect to item 
selection* Item assignment* and number of items per form. Results sup- 
port the appropriateness of the procedure for es^jlmatlng group achlevfcmant. 



OBJECTIVES 



The problem investigated was the feasibility of concurrently and 
randomly sampling examinees and items in order to obtain group data 
generalizable to a universe of examinees and to a universe of items* More 
specifically, the purpose of the study was to use matrix sampling to 
evaluate the ability of randomly selected populations of second and third 
grade children to solve simple equations derived from a + b • c and from 
c » a + b. 

The term "matrix sampling*' is used here to denote the concurrent 
sampling of M examinees from a universe of M examinees and N items 
from a universe of N mathematici 1 items. The M examinees were randomly 
partitioned into k mutually exclusive samples of m and the N test 
items were randomly partitioned into k non-overlapping samples of n items 
where M » km and N •=» kn. The k > 1 item samples were randomly assigned 
and administered to the k examinee samples. 

The terms "universe", "population" ar.d "sample" are used in a 
hierarchical sense* A sample is a subset of a population which in turn 
is a subset of a universe. 

The study d:fffered from prev^.ously reported matrix sampling studies 
(Sjogren, 1970) along several dimensions! (1) the item population was a 
proper subset of a well-defined item universe, (2) the sampled items were 
administered not in a larger testing context but as an Independent unit, 

(3) the examinee universe Included relatively Immature individuals in 
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terms of mathematical and chronological maturity, and (4) the data generated 
was used to estimate group performance and to establish group norm statistics* 

The specific questions considered in this report, a subset of a larger 
investigation (Montague, 1971) are the following: 

1. Within an examinee population at each grade level and 

1.1 for a given item population and a given test format, do 
examinee population (Mean estimated test scores) differ 
when systematic distribution of test items to forms is 
contrasted with random assignment of items to forms? 

1.2 for a given item universe and a given test format, do 
examinee population METSs differ when systematic selection 
and systematic assignment of items to forms is contrasted 
with random selection and random assignment of items to forms? 

1.3 for a given item universe and a given test format, do 

examinee population METSs differ when systematic selection / 

and random assignment of items to forms is contrasted with 
random selection and random assignment of items to forma? 

1.4 for a given item population, do examinee population METSs , 

differ when systematic distribution of test items to k forms 

is contrasted with random assignment of items to 2k forms? \ 

V 

1.5 for a given item universe, do examinee population METSs 
differ when systematic selection and systematic assignment 

of items to k forms is contrasted with random selection and 
random assignment of items to 2k forms? 

2. Across examinee populations at each grade level and 

2.1 for a given item population and a given test format, do 
examinee population METSs differ when systematic distribution 
of test items to forms is contrasted with random assignment 
of items to forms? 

2.2 for a given item universe and a given test format, do 
examinee population METSs differ when systematic selection 
and systematic assignment of items to forms is contrasted 
with random selection and random assignment of items to forms? 

2.3 for a given item universe and a given test format, do examinee 
population METSs differ when systematic selection and random 
assignment of items to forms is contrasted with random 
selection and random assignment of items to forms? 
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2*4 for a given item population, do examinee population METSs 

differ when systematic distribution of test items to k forms 
is contrasted with random assignment of items to 2k forms? 

2.5 for a given item universe, do examinee population METSs 

differ when systematic selection and systematic assignment 
of items to k forms is contrasted with random selection 
and random assignment of items to 2k forms? 

Methods 

Item Universe and Item Population : The item universe was the composite of 

two distinct domains: a domain of basic addition and subtraction facts and 

a domain of senctence types. Basic addition and subtraction facts whose 
sum was between 10 and 18 and whose addends were unequal comprised the 
domain of number facts. The domain of sentence types included simple 
equations in one unknown derived from a + b • c and c » a + b. The 
640-element item universe reflected a systematic asslgnmeiit of each of the 
number facts to each of the simple equations. 

Four 32-element proper subsets of the item universe were identified 
as item populations. Three of the item populations consisted of items 
randomly sampled from the item universe with inter-population item 
replacement and without intra-population item replacement. The fourth item 
population was identical to that used in a preexisting inventory (Weaver, 
1970). The item populations are identified in Table 1. 

Instruments : Seven test instruments reflecting one of four item populations 

were constructed. Test T7 consisted of items systematically drawn from the 
item universe and systematically assigned to four test forms each of eight 
items. The item population for T7 as well as the format was identical to 
that of the Weaver Inventory. The item populations for tests Tl, T2, and 
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T3 consisted of all items from the item population for T7 randomly 
assigned to four or eight or two forms of 8 or 4 or 16 items respectively. 

The remaining tests, T4, T5, and T6, consisted of items randomly sampled 
from the item universe and randomly assigned to four, or eight, or two 
forms of 8, or 4, or 16 items respectively. The structure for the seven 
tests is contained in Table 2. A sample test precedes the references for 
this report. 

Examinee Universe and Examinee Population . The examinee universe consisted 
of pupils using a common basal mathematics series from public schools in 
Madison, Wisconsin. From a listing of the 218 second and third grade 
classes, 79 classroom units were randomly selected. Across these units, 

/ 

12 examinee populations at each grade were identified. i 

At each grade level, six examinee populations took form 1, or form 2 ^ 

or form 3, or form 4 of the preexisting Inventory and form 1, or form 2, ... , 
or form k (k » 2, 4, 8) of an item sampled test. Also at each grade level, 
six examinee populations took form 1, or form 2, ... , form k of an 
item sampled test. The characteristics of the examinee populations are 
identified in Table 3. 

Data Source ; The data unit within each examinee population was the 
estimated test score for a given examinee within that population derived 
from performance on one of the forms of a given test. For a given test 
of k forms each of n items per form such that nk “ 32, the estimated 
test score for a given examinee was kn* where n* was the number of 
items correct. The mean estimated test score (METS) for a particular 
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examinee population was the average of the estimated test scores for that 
population* In subsequent analyses, the data unit was the population METS. 
i.e., the METS based on all pupils within a given population who (regardless 
of classroom unit) took a particular test (regardless of the form of that 
test). One way ANOVAs were used with the entry being the population METS, 

Table A contains the METS and the estimated test score variability 
within examinee populations for each test administered. 

Results ; No F-statistics were significant at the a « ,05 level of 
significance for questions 1.1, 1,5, 2,1 2,5 as considered in this 

report. The results are summarized in Table 5 in terms of specific contrasts. 

Educational Importance ; Subject to the constraints of the present / 

I 

Investigation the results suggest that adequate approximations of 

population METS were attainable for a variety of school examinee populations 

when the item universe and the examinee universe were well-defined and \ 

randomly sampled. Aspects of the feasibility include; 

1, The procedure is appropriate when administering a sample of items 
in a separate testing context rather than as part of a larger testing 
program, 

2. Wide diversity in test construction can be accommodated. The 
necessity for a single "standardized” instrument has been considerably 
reduced because comparable METSs can be obtained either by item sampling 
from an existing item population or from an encompassing Item universe and 
randomly assigning items to the same number of forms or to fewer forms. 
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3. The efficiency of matrix sampling in terms of the time unit per 
pupil per item was supported. Accurate estimates of group achievement for 
relatively young children can be obtained in half the time per pupil* The 
procedure was empirically efficient for test forms with as few as four 
items. 

4. To support or disclaim a contention that two groups were comparable 
with respect to selected, well-defined areas of content, alternative 
procedures have been provided. The alternatives provided for generalisation 
to a well-defined universe of items, or to the population of items, or to 
the well-defined universe of examinees. 
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TABLE 1 



Item Populations Used for Tests 
n, T2, T3, T4, T5, T6^ & T7 



Item Population Tests Tl, T2, T3, & T7* 


11 - X « 5 


X ■ 6 + 7 


13 “ X + 4 


7 » X - 6 


4 + 8 *» X 


12 - 9 « X 


8 + 9 « X 


11 ** 2 + X 


9 » X - 8 


X + 13 ■ 5 


X * 11 - 7 


14 + X » 8 


5 - 12 ■ X 


3 + X “ 11 


9 - X « 14 


15 - X “ 6 


16 - 9 + X 


9 - 11 - X 


6 + X s* 11 


X + 3 - 12 


X + 4 « 11 


X ■" 7 “ 8 


5 - 12 + X 


13 *» 5 - X 


9 ■ X + 14 


15 “ X + 6 


X - 9 - 7 


X *» 7 + 8 


X « 13 - 9 


X - 8 - 14 


8 « 12 - X 


11 - 3 - X 



Item Population Test T4** 


X ■* 6 + 7 


13 » 7 + X 


X - 13 - 5 


12 


- X - 7 


8 + 3 « X 


5 » X - ■ •9 


X + 13 - 9 


11 


» 9 - X 


5 - 12 - X 


3 * X - 9 


X - 5 + 9 


8 » 14 + X 


X - 6 « 5 


8 X *» 15 


X + 11 «• 2 


14 


- X + 8 


13 ** X + 4 


2 + X " 11 


X - 6 * 8 


9 • 


- 15 » X 


7 - X * 13 


X ■ 9 + 3 


7 " 16 - X 


X “ 3 + 8 


X *» 11 - 8 


2 9 * X 


X «■ 11 - 6 


X *■ 7 + 8 


12 + X “ 3 


14 + X “ 5 


15 - 9 "■ X 


14 


X + 9 


Item Population Test T5** 



X - 7 - 8 
13 - 9 - X 
7 - X « 11 

4 » X - 8 
7 + 6 - X 

12 - 5 - X 
7 " X - 8 

13 - 9 + X 



3 - X - 9 

3 - X - 12 

11 - X - 9 

4 + 8 «• X 

16 - X •* 9 
X - 16 - 9 
9 - X + 17 

X + 4 * 13 



9 “ X + 14 
15 - X 6 
X - 8 " 5 
15 • X + 8 
X » 11 - 4 
9 - X ■ 14 
8 - X - 7 
X - 9 + 3 



12 - 4 + X 
14 - X " 9 

7 » 16 - X 
11 - 4 - X 
9 + X « 15 
9 - X + 13 
5 - 11 - X 

8 • 13 + X 



*ltems Identical to those In preexisting inventory. 

**T terns sampled from item universe. 
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TALLE 1 continued 



Item Populations Used for Tests 
Tl, T2, T3» T4. T5, T6, & T7 



Item Population Test T6** 



X • 7 + 4 
X + 7 ■ 11 
6 - X 14 
15 - 7 - X 
3 • X + 11 
X + 7 - 13 
X • 11 - 5 
11 - 9 - X 



8 • 14 - X 
X - 7 •• 8 

9 + 3 • X 
9 - 11 - X 

12 » X + 8 
X - 3 *■ 8 

8 + X - 12 

13 - X • 6 



X 4* 4 ■» 11 
4 - X • 12 
X + 8 - 17 
X “ 3 + 9 

7 - X • 15 
6 - 13 - X 

8 • 14 4* X 
11 - 7 4- X 



16 « 9 4- X 
X - 6 • 5 
X 4- 4 • 13 
11 • X 4- 8 
5 " X - 8 
X 4- 9 - 13 
X - 5 - 9 
15 4* X • 7 



TABLE 2 

Summary of Distinguishing Characteristics 
Of Tests Tl, T2, T3, T4, T5, T6, & T7 




/ 



I 



V' 




10 



TABLE 3 










d 






0 






•H 








€ 




d 


fS 










•H 






U 




4J 






CO 




•g 


•H 


CO 


0 


P 


a 






o 






•H 


d 


V 




c3 


(0 


(0 




pH 


(U 


d 


fl *o 


o 




(Cl 


•H 


o 


M 




(Xi 


U 


CO 






o 


a 


M 


•H 


(U 


O 




a 


ft* 


•H 


*5l 




4J 






S 


s 










M 


(H 






o 





sf 

* •> \0 

VO 


1 

F6: 

55/53 


T6: 

2 forms 
16 items 
each 


Items Sampled From 
Item Universe 

1 - - ■ - - i 


•• S 

W 

vO 


F5: 

58/59 


T5; 

8 forms 
4 items 
each 

< ! 


<r 

•• VO 

<r ^ 

M ON 

m 


[ F4; 

57/63 


T4; 

4 forms 
3 items 
each 


E3: 

63/61 


F3: 

53/52 


T3; 

2 forms 
16 items | 
each 

1 

1 


Items Sampled From 
Preexisting Inventory* 


o 

• • fSi 

OJ 

M vD 


F2; 

63/63 


T2; 

8 forms 
4 items 
each 


i 

El; 

60/62* 


FI: 

56/63 


i § 

• « 

H 0 .£3 

H (M •H ^ 

(0 

Sf 00 0) 



vo 



CO H 






8 

»4 

o 



in 

H 



•<T 

H 



H 




*Tbe first numeral In each cell of the matrix indicates the number of second grade pupils in a particular 
population. The second numeral indicates the nuzober of third grade pupils. 



TABLE 4 



Mean Estlsiated Test Scores and Estlioated Test Score 
Variability Across Examinee Populations 



Test 


Number 


Grade 


s 2 


Grade 3 


of 

Test 

Forms 


Mean 

Estimated 
Test Score 


Estimated 
Test Score 
Variability 


Mean 

Estimated 
Test Score 


Estimated 
Test Score 
Variability 






Examinee Population E 


Tl 


A 


19.79 


8*08 


23*61 


5*96 


T2 


8 


22.10 


9*16 


23*08 


7*94 


T3 


2 


20.82 


6*11 


23.60 


3*17 


T4 


4 


21.29 


6*03 


23*18 


5*25 


T5 


8 


18.94 


9*29 


22*42 


7*31 


T6 


2 


18.76 


8*40 


24*03 


5*80 


T7 


4 


19.99 


6*52 


22*99 


5.94 




Examinee Population F 


Tl 


4 


19,21 


8.26 


22*85 


7.65 


T2 


8 


20.69 


8.86 


23*61 


8.48 


T3 


2 


20*45 


7.54 


22*96 


6.74 


T4 


4 


19*64 


7.83 


21*58 


7.69 


T5 


8 


17*92 


8.18 


21*83 


9.06 


T6 


2 


21*16 


7.12 


21*54 


5.80 



Note.— The mean estimated test score for a particular examine® population was 



n 






where la the proportion correct for 
examinee J on one of the k forms of test 
T1 (1 1 A 1 7)» 

and n Is the number of examinees within 
A particular population taking a form of 
Tl. 



The estimated test score variability Is the standard deviation of the distribution 
of the estimated test scores for the particular sample defined by the parameters of 



a particular contrast* 
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TABLE 5 



Summary of Contrasts and 
Significance Levels 



Question* 


Contrast 


Significance Level of Computed F 


Grade 2 


Grade 3 


Within Examinee Population E 


1.1 


y(T7) - y(Tl) - 0 


P > 


.05 


P > 


.05 


1.2 


U(T7) - y(T4) « 0 


P > 


.05 


P > 


.05 


1.3 


]j(Tl) - y(T4) « 0 


P > 


.05 


P > 


.05 


1.4 


y(T7) - W(T2) » 0 


P > 


.05 


P > 


.05 


1.5 


p(T7) - y(T5) - 0 


P > 


.05 


P > 


.05 




Across Examinee Populations E 


and 


F 




1.1 


y(T7,E) - W(T1,F) - 0 


P > 


.05 


P > 


.05 


1.2 


y(T7»E) - y(T4,F) » 0 


P > 


.05 


P > 


.05 


1.3 


y(Tl,F) - y(T4,F) » 0 


P > 


.05 


P > 


.05 


1.4 


y(T7,E) - y(T2,F) » 0 


P > 


.05 


P > 


.05 


1.5 


y(T7,E) - y(T5,F) « 0 


P > 


.05 


P > 


.05 



*See pag..s 2 and 3 of this report. 
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My name 



Grade 



a. 



b. 



c. 



d. 



ERIC 



is 



School 



012**H456789 
10 11 12 13 14 15 ■ 17 18 19 



NUMBER PUZZLES 




• I 

9 + B = 10 

H = 3 H- 5 
0 - 4 » ■ 
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